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Abstract 

This paper presents an interior design framework based on 
multi-touch table, which allows multiple users to work on. 
The information visualization is enhanced by 3D animation 
shown in a vertical display using the mixed-reality 
technology. Furthermore, the framework is extended to 
allow both collocated and remote collaboration, which is 
enabled by the combining of separated viewed mode and 
extended view mode. Moreover, a "sliding pass" touch 
gesture is designed for the extended mode, and a user avatar 
system is developed to solve the user awareness problem. 
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Introduction 

We are interested in both co-located collaboration 
[Hornecker 2008, Tobiasz 2009, Sadurai 2009, Cao 2010] 
and remote collaboration [Arroyo 2010, Tuddenham 
2007, Perron 2006, Coldefy 2007] between groups 
using the large-scale multi-touch tabletop. Large 
multi-touch tabletop wall and displays is one of the 
most popular approaches devised for computer- 
supported cooperative work (CSCW) [Grudin 1994], 
since it offers intuitive interfaces for the team members 
to manipulate both shared and individual instances of 
data representations concurrently. 

One of the most important factors in the 
implementation of multi-touch tabletop system for 
CSCW is the information visualization method. One of 
the related outstanding projects is Lark, presented by 
Matthew [Tobiasz 2009], an information visualization 
system for hierarchical data that supports co-located 
collaboration by integrating a representation of the 



information visualization pipeline in the shared 
workspace. However, many of the currently 
developed multi-touch tabletops confine its 
information visualization technology within 2D, which 
makes it difficult for workers to complete their work 
purely via the tabletop, especially for some design jobs. 
There are also some 3D visualization methods for 
multi-touch tabletop, for example, the FI3D project 
presented by Yu [Yu 2010]. However, most of them 
only support for single user instead of collaboration. 

Besides, the physical constraint of most multi-touch 
tabletops may also bring problems for collocated 
collaboration. A few recent projects have investigated 
the possibility of physically extending the LCD based 
multi-touch tabletops by combining a few of LCD 
panels together to increase the physical size of the 
system such that more users can work together [Kim 
2009, Wang 2008]. However, there are also some 
limitations with these approaches. For example, the 
boundaries of each LCD display explicitly divide the 
whole surface into small territories, which is not quite 
helpful for enforcing the integrity of the whole 
workspace. The boundaries may also cause problems 
when user's touch operations are across the 
boundaries. 

As for remote collaboration, several recent projects 
have investigated the possibility of building linked 
connections among geographically-separated 
tabletops to create a shared. For example, 
VideoWhiteboard [Tang 1991] and Clearboard [Ishii 
1992] provided shared drawing surfaces to let users at 
different location simultaneously sketch objects. 
Escritoire [Ashdown 2005], RemotedDT [Esenther 
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2006] and ViCAT/TIDL [Hutterer 2006] provide 
remote collaborators with another shared workspace 
with movable interactive virtual objects so that remote 
collaborators can arrange the objects at different 
location but still get the same experience of collocated 
collaboration. Although Distributed Tabletops [Tang 
2006] have improved the previous work and virtual 
objects can be moved and reoriented by any 
collaborators to suit different seating arrangement, the 
whole shared workspace is still relatively static to 
collocated users. User operations could only happen 
on individual objects. In reality, this approach may not 
always be suitable for remote collaborations because 
the collaborations task may be divided into different 
sub tasks and each collocated group can choose to 
focus on a certain sub-area of the whole shared 
workspace. Thus, user operations should be 
implemented to support local view manipulation. 

Another important factor to ensure the effectiveness of 
remote collaboration is user awareness. This often 
means to provide different kinds of cues or indicators 
to remind a user of the physical existence of the 
remote collaborators. Embodiment of collaborator is 
an intuitive way to represent collaborator's activities 
in the reference space and to enforce user awareness. 
Recent researches show a high interest in video-based 
embodiments of collaborators. In Tang's Video Arms 
[Tang 2006], the image of collaborators' arms is 
captured digitally and redrawn at the remote location 
to mimic collaborators' body gesture. Several other 
projects [Pauchet 2007, Izadi 2007, Tang 2010] also 
adopt this method to enable remote user awareness. 
These researches all assume that only one participant 
at each workstation uses the remote collaboration 
systems. However, this may not always be true in 
reality. For example, in a design team that includes 
many designers located at different cities, one designer 
may occupy a single tabletop at one location whereas a 
few designers at another location may want to share a 
tabletop together. Tang's research [Tang 2010] also 
shows that attributing workspace activity to a remote 
collaborator is difficult when there is more than one 
remote collaborator and participants are sometimes 
confused about whom the hand belongs to when all 
collaborators have the same-side configuration. 

In this article, we present a conceptual interior design 
frame-work - multi-touch mixed reality (MTMR) 
which integrates the usage of mixed reality (MR) 
[Milgram 1994] with the multi-touch tabletop interface, 
providing 3D spatial information to the designers. 
Moreover, the proposed MTMR system is extended to 



a networked collaborative interior design system with 
the client-server architecture in order to support both 
efficient collocated and remote collaborations. Besides, 
two modes of view - Separated View Mode (SVM) 
and Extended View Mode (EVM) - are implemented. 
The SVM is aimed to address the physical constraint 
problem in co-located collaboration and the EVM is 
aimed to provide independent views of shared 
workspace in remote collaboration. We also develop a 
user avatar system to improve user awareness in 
remote collaboration. 

Interior Design Framework 




R view of 
the tabletop 



c. Top view 
ispiayed on 
the tabletop 



FIG. 1 OVERVIEW OF THE DESIGNER SIDE 




FIG. 2 OVERVIEW OF THE CLIENT SIDE 

The MTMR design framework is divided into designer 
side (Fig. 1) and client side (Fig. 2). At the designer 
side, designers interact with the tabletop, and carry 
out design work on the top view of their design. 
Vertical display reflects the real-time 3D simulation of 
design perspective view, which provides designer 
better vision of their design. At client side, client 
resides at his/her remote home and receives designers' 
control messages. Client's computer will be able to 
interpret these control messages to construct the 
virtual view of the design output. The design output 
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which includes furniture and their orientation is 
combined with the camera-captured video to establish 
MR view. Detailed interactions at both sides are 
discussed in following sections. 

Designer Side 

The major parts of the designer side are the tabletop 
interface (Item c in Fig. 1) and the vertical display 
(Item b in Fig. 1). The tabletop interface works as both 
input and output device, while the vertical display 
shows the corresponding 3D MR view. The tabletop 
interface provides a multi-user multi-touch supported 
platform for collaborative work. Since it is convenient 
for the designers to carry out furniture arrangement 
work on 2D top views, the tabletop interface is chosen 
to provide the interactive environment. However, a 
top view alone may not provide 3D spatial 
information for the designers, thus the 3D MR view on 
the vertical screen can deliver complementary 
information. 

Since the top view is limited in 2D, a 3D MR view of 
the design can provide important complementary 
information to the designers to review their work. The 
position, orientation and size of the virtual furniture 
contained in the tabletop display are passed through 
network to the computer connecting the vertical 
screen, where they are interpreted to construct and 
register 3D virtual furniture models. A camera 
mounted on top of the vertical screen (Item a in Fig. 1) 
captures live video of the design work going on the 
tabletop. This real-time video is mixed with the 
constructed 3D furniture models in such a way that 
these furniture models in the MR view align with their 
position, rotation and size seen in the top view. The 
final output is these 3D models appearing to stand on 
the tabletop (Fig. 4 and item b in Fig. 1). What is more, 
since the designers' operation is also captured in the 
video, they may look at the vertical screen while 
operating on the tabletop. This extra feature is 
expected to make the designers have the feeling that 
they are interacting with the 3D models directly, 
which would narrow the gap between the physical 
world and the virtual world. 

Fig. 3 shows the GUI of the application at the designer 
side. Besides, several traditional touch gestures 
(translation, rotation, scaling) are implemented for 
touch operations. In summary, the application at the 
designer side supports functionalities in the following 
three aspects: 

■ Furniture model manipulation: The system 



provides the Model Catalogue and Trash Bin 
widgets for creating/deleting furniture models. 
Two one-finger gestures can be recognized for 
moving/sliding the models around inside the 
shared workspace and three two-finger 
gestures can be recognized for translating, 
rotating and scaling the models 

■ Workspace manipulation: In the shared 
workspace, a one-finger gesture can be used to 
move the entire workspace to any direction 
inside local workspace view. Three two-finger 
gestures can be used to translate, rotate and 
scale the entire workspace. In addition, a 
switchable mini-map widget is also designed 
for showing the area of the workspace that is 
visible in the local workspace. 

■ Two switchable collaboration modes: The 
system also provides both separated view 
mode (SVM) and extended view mode (SVM) 
to facilitate different collaboration needs which 
are switchable using the View Mode Switch 
widget. More details about these two modes 
will be provided later when collocated 
collaboration and remote collaboration on the 
MTMR system are introduced. 




FIG. 3 THE DESIGN OF GUI AT THE DESIGNER SIDE 



The development tool used here is MXR software 
development kit (SDK) which is the product of MXR 
Corporation (http://www.mxrcorp.com/). This SDK is 
built on the Torque Game Engine Advanced which is a 
popular commercial 3D game engine. Thus, the 3D 
virtual world simulation is mainly handled by TGEA 
source code. The modification is mainly done to 
facilitate creating MR effects. 

The first part of 3D simulation implementation is 
importing 3D models for use (See Fig. 4 for example). 
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Furniture models used in this project are from the 
model database of another interior design software. 




FIG. 4 PERSPECTIVE VIEW OF A FURNITURE MODEL SAMPLE 

Since TEGA has its own static 3D data format dts, and 
the available models are in 3ds format, the first step of 
importing them is transferring the format using 
related software tool. When creating furniture models, 
properties such as size, location and orientation are 
retrieved from data sent from designer side to 
configure them. 



The second part of 3D simulation implementation is 
collaboration. 




FIG. 5 CALIBRATION INTERFACE 




FIG. 6 CALIBRATION MARKER 

Fig. 5 above shows the calibration interface. To finish 
the collaboration, a marker with pattern shown in Fig. 
6 needs to be captured into the green ring. Then the 
centre of the marker will be labeled as origin (0, 0, 0), 
x-y plane is aligned with the marker's surface plane, 
and z axis point from the origin upwards. 

For vertical display at designer side, marker is shown 
in the center of tabletop graphical interface. After 
collaboration, the marker is removed and the tabletop 



surface will be x-y plane. Thus, when z value of 
furniture model's coordinate is set to 0, the furniture 
will appear to be held on the tabletop as shown below. 

For client side 3D simulation, the marker is placed at 
the center of the room floor such that the floor forms x- 
y plan. In this way, furniture models will appear to 
stand on the floor of the physical room. 

After the coordinate system is set up, it is rather easy 
to match the top view design on tabletop with the 3D 
simulation at both vertical display and client side. 

Client Side 

At the client side, the implementation of MR (Item g in 
Fig. 2) is similar to the 3D MR view at the designer 
side. The client resides at his/her remote room and 
receives designers' control messages. These control 
messages, including the existence, position, orientation 
and size of the virtual furniture in the design plan, are 
passed to the client's computer through network (Item 
/ in Fig. 2) to construct and update the client's MR 
view in real time. A camera (Item d in Fig. 2) captures 
live video of the client's room. This video is 
augmented with life-size virtual furniture models 
constructed according to the control messages. The 
difference is that at the designer side, the camera is 
fixed, while at the client side, the client is supposed to 
view the furniture layout from different perspectives 
with a rotatable camera. To match the perspectives of 
the virtual world and the physical world, an extra 
gyroscope (Item e in Fig. 2) is attached to the camera to 
measure its orientation. The measurement data is used 
to adjust the view point in the virtual world. Fig. 7 
shows the same design (as in the top view in Fig. 1) 
viewed from three different perspectives at the client 
side by rotating the camera from left to right. 

Collocated Collaboration and Remote 
Collaboration Using MTMR System 

In the case of interior design, when the room is large, 
it needs more than one designer to collaborate. When 
the team of designers is large, the MTMR system with 
only one tabletop is not enough. It is important for the 
MTMR system to support distributed tabletops for 
collocated collaboration when the team of designers 
are in the same place and remote collaboration when 
they locate in different places. As mentioned in the 
introduction, the limited physical size is often a reason 
to cause unpleasant experience when too many 
collocated collaborators crowd around the same 
tabletop. As for remote collaboration, user awareness 
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is an important factor to ensure the effectiveness of the 
collaboration. 




(A) 




(B) 




(C) 

FIG. 7 (A)-(C) SHOW THE DESIGN AT THE CLIENT SIDE FROM 
THREE DIFFERENT PERSPECTIVES BY ROTATING THE 
CAMERA FROM LEFT TO TOP 



Motivated by these problems, the proposed MTMR 
system is extended to a networked collaborative 
interior design system with the client-server 
architecture to allow efficient collocated and remote 
collaborations. The system provides an independent 
shared workspace for each collaboration task. 

Network Architecture 

Fig. 8 shows the architecture of the networked MTMR 
system. The overall system consists of a single server 
application named "SAPP" running on an "always- 
on" PC with known IP address and several instances 
of the client application named "CAPP" running on 
several workstations located at different places. At 
each workstation, the client application runs on the 
multi-touch tabletop and the augmented 3D 
application named "3DAPP" runs on the vertical 
display. 




FIG. 8 NETWORK ARCHITECTURE OF THE DISTRIBUTED 
MTMR SYSTEM 

In this networked system, the server application is 
designed as a data synchronization manager and 
communication coordinator. It is mainly responsible 
for storing project information, keeping track of 
session information and directing messages among 
different clients. For each project, it maintains an 
individual file keeping records of the project meta- 
data and project scene data. Meta-data includes 
information that will help server application initialize 
a new session. Project scene data mainly consists of a 
list of data structure that represents all objects in the 
scene. The major attributes that are used to describe 
each object are object index, type, position, rotation 
and scale. 
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Separated View Mode and Extended View Mode 




(B) TABLE 



FIG. 9 A SCREENSHOT OF THE CLIENT APPLICATION 
RUNNING IN SVM 

The system also provides two different collaboration 
modes to facilitate different collaboration needs, which 
is enabled switchable by the View Mode Switch 
widget. 

In Separated View Mode (SVM) (See Fig. 9), each 
multi-touch tabletop can have its own view range of 
the shared workspace. Workspace manipulation on 
tabletop A does not affect the view range of tabletop B. 
But furniture model manipulation on tabletop A is 
synchronized with the shared workspace and is 
reflected at other tabletops. In order to keep the 
integrity of the furniture models in the shared 
workspace, each model can only be manipulated by 
the users at the same tabletop. This mode is suitable 
for most remote collaboration tasks. 

In Extended View Mode (EVM) (See Fig. 10), two 
multi-touch tabletops are virtually concatenated 



together so that a larger single multi-touch tabletop 
surface can be constructed just by putting the two 
tabletops side by side. Both workspace manipulations 
and furniture model manipulations are synchronized 
with each other. Workspace and all objects inside the 
shared workspace can be manipulated by users at 
different tabletops at the same time. This mode is the 
most suitable for overcoming the physical constraint 
of collocated collaboration. 




(B) TABLE 2 



FIG. 10 A SCREENSHOT OF THE CLIENT APPLICATION 
RUNNING IN EVM 

A New Touch Gesture for Collaboration in EVM 

Assume that a team of designers are collaborating in 
EVM, and one designer working on one tabletop 
wants to pass an object to another designer working 
on another tabletop. Traditional touch gestures cannot 
achieve this. Thus, a sliding gesture is implemented to 
improve user experience when they are collaborating 
in EVM. 

We achieve this in three steps. Firstly, the simple 
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translation gesture is recognized, and the speed of the 
touched item is estimated. Whenever a translation 
happens on the item, the intrinsic speed is estimated 
by dividing the displacement with time difference. 
The new estimated speed is calculated by taking the 
average of the intrinsic speed and the previous 
estimated speed. Secondly, when the finger is 
removed from the item, the estimated speed is then 
used as the start value of the sliding animation. The 
animation is played to indicate the sliding operation 
and the speed is reduced in a quadratic curve. Finally, 
when the object is passing through the boundary of 
one tabletop and entering another, the server 
application plays data synchronization, simulating 
that the object is seamlessly passing between two 
virtually concatenated tabletops. 



User Avatar System 




FIG. 11 A SCREENSHOT OF THE USER AVATAR SYSTEM AT 
TABLE 1 



In order to solve the user awareness problems in 
remote collaboration, a user avatar system enabling 
following functionality is implemented (See Fig. 11): 

■ A list widget for showing the icons of all 
workstations that have joined in the current 
project. 

■ A list widget for displaying the avatars of all 
participants that are registered on a selected 
workstation. 

■ A mini view widget for displaying the remote 
workspace view of a selected remote 
workstation. 

■ It is able to display a user's avatar on the object 
that the user is currently touching on. 

■ It provides a short animation to enlarge the 



user avatar of an object that a user touches in 
the mini view widget. 

The main benefit of the user avatar system is that it 
allows a user easily recognizes the existence and 
manipulation of another remote user without having 
any interfering effects on this user's own manipulation. 

Example User-Case Scenario 

MTMR system is a conceptual design framework that 
utilizes an enhanced information visualization method 
(3D MR) to improve users' experience on indoor 
spatial information rendering, which is believed to be 
helpful for interior design. Besides, the application at 
client side provides an opportunity for the client to 
view the temporary design easily and intuitionally. 

Moreover, the extension of the MTMR system for 
collocated and remote collaboration is aimed to solve 
the problems such physical limitation of single 
tabletop and remote user awareness which are 
common in currently developed multi-touch tabletop 
based CSCW projects. 

To show the use of our system, we describe a scenario 
in which a large designer team, whose members are 
located in different places, is carrying out an interior 
design for a remote client. We will explain how our 
design features are used as we follow how the 
designer team carries out the design work. 

Communication between Designer and Client 

Suppose that a designer team works in Singapore, 
carrying out an interior design job for a client in China. 
The client needs the design team to send the detailed 
information about the room, such as its height, width 
and shape. Using the proposed MTMR system, 
members of the designer team can do the job by co- 
located collaboration or remote collaboration on the 
distributed tabletop (introduced later). After the initial 
design is completed, the design team sends the client 
the information about the design, and then, the client 
can use the proposed application at the client side to 
view the vivid live video of the current design on his 
laptop, which offers the client an opportunity to give 
instant and helpful feedback. 

3D MR 

During the design, the designer may need to take into 
account the effects of the furniture's height played on 
the room. Without the vertical display in our system, it 
is difficult for the designers to achieve this because 
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most tabletop displays can only provide 2D 
information (the height and the width). Although by 
adding a rotation gesture and the relative object 
rendering with respect to the slant and tilt direction, it 
is possible to implement 3D simulation in the tabletop 
display, this newly added gesture will make 
impossible the collocated collaboration on a single 
tabletop or distributed tabletop in EVM mode. This is 
because certain designer's manipulation using the 
slant and tilt gesture will always distract others who 
share the same workspace from their work. 

Thus, the vertical display and its 3D MR simulation is 
a reasonable way for the designer to view the 3D 
spatial information of their design. It won't lead to any 
distraction to others, since the rendering of the 
perspective view has no effects on the manipulation 
workspace. 

Collocated and Remote Collaboration 

Suppose that there is the designer team includes six 
designers, four in Singapore are in charge of placing 
the furniture; another two in Hong Kong are in charge 
of paining the floor. The four in Singapore need two 
tabletops, one displays the eastern part of the room 
and the other western part. These two tabletops are in 
EVM which enlarges the workspace to allow them to 
work concurrently. If one designer finds that the 
furniture object which is under manipulation of his 
colleague, is also suitable for the space he is in charge 
of, he can ask his colleague to pass the object via the 
sliding gesture. 

The two in Hong Kong and in charge of painting floor 
only need one tabletop. Since the placing of furniture 
has great effect on the final outlook of the floor, they 
need to remotely collaborate with the four in 
Singapore. However, instead of EVM, SVM is 
activated in the tabletop because they need a whole 
view of room to efficiently paint the floor. Sometimes, 
the manipulation of the furniture may distract their 
painting. When this happens, the user avatar system 
provides them with information about who are 
manipulating the distracting furniture so that they can 
call the corresponding designer in Singapore to stop 
for a while via the speech communication system. 

Conclusions 

We have developed MTMR, a conceptual interior 
design framework which integrates MR with the 
multi-touch tabletop interface, to provide an intuitive 
and efficient interface for collaborative design and an 



augmented 3D view to users at the same time. 
Moreover, we have extended the MTMR system for 
both collocated and remote collaboration. 
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